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Greetings  from  Ms.  Mary  J.  Miller,  Principal  Deputy  Assistant  Secretary  of 
Defense  for  Research  and  Engineering 


Our  near-peer  adversaries’  investments  in  research  and  development  have  increased 
dramatically,  as  a  result,  the  United  States  is  no  longer  able  to  boast  of  technology 
superiority.  Our  adversaries  watched  us  for  years  and  saw  our  equipment,  studied  our 
tactics,  techniques  and  procedures,  and  determined  our  Concepts  of  Operation.  They 
are  investing  in  technology  and  capabilities  attempting  to  avoid  our  strengths  and  exploit 
our  weaknesses.  To  maintain  our  technological  edge,  our  scientist  and  researchers  must 
work  together  to  develop  new  and  advanced  capabilities  and  technologies. 

Technology  changes  at  great  speed;  therefore,  we  must  continue  to  push  the  limits 
and  work  with  our  stakeholders  -  academia,  industry,  Congress,  and  our  allies.  We  face 
the  competing  necessities  of  protecting  our  sensitive  research  and  collaborating  with 
partners  outside  of  the  DoD.  Our  scientists,  researchers,  and  engineers  must  continue 
to  have  the  ability  and  opportunity  to  gain  peer-review  status  for  limited  or  classified 
projects,  and  to  share  information  and  further  leverage  discoveries. 


In  an  effort  to  ensure  an  alternative  peer-review  framework  exists  for  the  research 
and  engineering  enterprise,  we  initiated  a  peer-reviewed  journal  forum  to  afford  our 
scientists  and  technologist  an  opportunity  to  share  their  research  in  a  controlled  access 
environment.  The  Defense  Technical  Information  Center  (DTIC)  developed  an  internal 
journal,  Journal  of  DoD  Research  and  Engineering  (JDR&E),  to  recognize  the  science 
and  technology  community’s  best  works  at  the  Controlled  Unclassified/Classified  level. 


Through  its  print  and  online  incarnations,  JDR&E  aims  to  publish  influential  papers 
to  significantly  advance  scientific  understanding.  The  JDR&E  selections  present  novel 
and  broadly  important  data,  syntheses,  or  concepts.  They  should  merit  recognition  by 
the  wider  DoD  science  and  technology  community  and  address  the  need  to  collaborate, 
recognizing  contributions  from  our  research  community,  while  reducing  the  information 
available  to  the  public  and  the  potential  for  exfiltration. 


The  JDR&E  welcomes  submissions  from  all  fields  of  DoD  S&T.  The  editors  are 
committed  to  promptly  evaluating  and  publicizing  submissions  while  upholding  high 
standards  that  support  reproducibility  of  published  research.  The  JDR&E  will  be 
published  semiannually  and  selected  controlled  unclassified  publications  will  be  available 
in  print,  while  classified  publications  will  be  available  online. 


The  JRD&E  is  a  critical  piece  of  the  U.S.  retaining  technological  superiority  over  our 
adversaries.  We  must  all  work  together  in  order  to  stay  ahead  and  promote  the  exchange 
of  data  and  concepts  vital  to  the  Department  of  Defense  in  the  years  ahead. 
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Letter  to  Our  Readers  from  the  Principal  Director,  Research,  in  the  Office  of  the 
Assistant  Secretary  of  Defense  (Research  and  Engineering),  Mr.  Dale  A.  Ormond 


I  am  delighted  to  welcome  you  to  the  Journal  of  DoD  Research  and  Engineering 
( JDR&E) .  This  new  addition  to  our  family  of  printed  and  digital  peer-reviewed  publications 
will  foster  and  facilitate  our  collaboration  across  the  S&T  Enterprise,  including  the  17 
Communities  of  Interest  (Col).  Like  many  worthy  endeavors,  the  journal’s  foundation 
is  rooted  in  a  simple  goal:  to  encourage  our  researchers  to  publish  their  classified  and 
controlled  unclassified  (C/CU)  works  and  to  ensure  that  the  experts  on  our  Cols  who 
selflessly  volunteer  to  serve  as  reviewers,  validate  their  works. 

As  the  Principal  Director  for  Research,  I  have  oversight  of  all  of  the  DoD’s  S&T 
investments,  and  lead  the  Reliance  21  planning  effort.  Most  importantly,  my  job  is  to 
ensure  that  the  long-term  strategic  direction  of  the  Department’s  S&T  programs  will 
continue  to  generate  extraordinary  innovations  to  maintain  our  Military’s  operational 
and  technological  superiority  to  support  our  warfighter.  My  job  depends  on  the  expertise 
of  people  like  you,  the  researchers  and  engineers  who  work  in  our  DoD  Labs  and 
serve  in  our  research  communities.  For  a  long  time,  you  have  shouldered  tremendous 
responsibility  on  behalf  of  the  warfighter  and  continued  to  take  on  the  task  of  keeping 
them  safe.  However,  due  to  the  sensitivity  of  your  work,  you  have  not  been  able  to  gain 
the  recognition  that  you  deserve  for  your  efforts.  I  hope  to  inspire  you  to  take  advantage 
of  this  unique  opportunity  to  submit  your  exciting  work  to  this  journal.  We  want  to  give 
you  the  chance  to  highlight  the  support  you  have  received  from  your  organization  over  the 
years  and  recognize  you  for  your  accomplishments.  The  Journal  of  DoD  Research  and 
Engineering  is  here  to  provide  a  secure  venue  for  you  to  publish  your  work,  to  facilitate 
your  connection  to  the  larger  DoD  R&E  communities,  to  promote  your  collaboration  in 
cross-cutting  research  areas,  and  to  enable  you  to  share  and  use  the  Defense  Technical 
Information  Center  (DTIC)  repository  of  linkable  scientific  and  technical  information. 
To  all  the  DoD  Laboratory  and  Warfare  Center  Directors  who  are  making  good  use  of  the 
Section  219  Authorities  authorized  in  the  Duncan  Hunter  National  Defense  Authorization 
Act  for  Fiscal  Year  2009,  you  now  have  a  venue  by  which  you  can  meet  the  publishing 
requirements. 


From  the  silent  distance  of  space  where  your  robots  roam  and  service  our  satellites,  as 
Dr.  Roesler  expressed  so  eloquently  in  his  op-ed  article,  to  the  solitude  of  your  lab  where 
your  research  efforts  begin,  or  wherever  you  are,  I  encourage  you  to  publish,  volunteer, 
and/or  serve  as  the  journal’s  reviewers.  Reach  out  to  the  larger  communities  of  R&E 
to  collaborate  and  coordinate.  Our  warfighters  depend  on  you  to  continue  to  climb  to 
the  summit  of  your  field  and  do  the  impossible.  I  wish  you  all  great  success  with  your 
important  research  as  you  work,  publish,  and  help  us  ensure  the  S&T  we  continue  to 
pursue  is  well  beyond  any  other  nation  on  Earth. 
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Letter  from  the  Administrator  of  Defense  Technical  Information  Center, 
Mr.  Christopher  E.  Thomas 


I  am  excited  to  welcome  you  to  the  inaugural  issue  of  the  Journal  of  DoD  Research  and 
Engineering  (JDR&E).  During  my  tenure  at  the  Defense  Technical  Information  Center 
(DTIC),  I  have  noted  the  need  for  a  venue  that  provides  our  researchers  and  scientists  the 
capability  to  share  their  work,  while  still  affording  the  protections  needed  to  maintain  the 
competitive  advantage  our  S&T  community  provides  us.  The  JDR&E  is  a  peer-reviewed 
journal  of  DoD  classified  and  controlled  unclassified  (C/CU)  articles  that  cover  the 
scientific  and  technical  research  conducted  within  the  Department.  Under  the  guidance 
of  Ms.  Mary  Miller,  Principal  Deputy  Assistant  Secretary  of  Defense  for  Research  and 
Engineering,  and  the  management  by  DTIC,  the  JDR&E  will  serve  as  a  unique  platform  to 
encourage  collaboration  and  coordination  between  all  17  Communities  of  Interest,  from 
Advanced  Electronics  to  Biomedical  to  Space,  to  meet  the  DoD’s  S&T  challenges  of  today 
and  tomorrow.  The  JDR&E’s  key  functions  are  to: 


•  Recognize  researchers,  scientist,  and  engineers  doing  exceptional  work  at  the  C/CU 
level  and  provide  a  secure  platform  to  share  their  research 

•  Increase  DTIC  repository  capability  to  receive,  preserve,  manage,  and  disseminate 
scientific  and  technical  knowledge  generated  from  the  best  DoD  S&T  investments 

•  Promote  multi-agency  collaboration  in  cross-cutting  R&E  and  serve  as  a  pipeline  for 
DoD  future  breakthroughs 

DTIC  will  manage  the  JDR&E’s  semiannual  publication  to  ensure  all  C/CU  R&E 
articles  are  validated  via  a  peer-review  process.  The  JDR&E  is  a  venue  for  researchers 
across  DoD  to  publish  their  sensitive  work  and  avoid  information  exfiltration,  a  possible 
risk  often  associated  with  the  publication  of  data  in  academic  journals.  DTIC’s  security 
protocols  will  protect  against  this  threat— ensuring  the  right  balance  between  the  need  to 
share  S&T  activities  while  protecting  this  body  of  work  from  our  adversaries.  We  have 
many  excellent  researchers  in  our  DoD  Labs,  and  this  journal  provides  a  structured, 
secure,  and  premier  venue  to  publish  their  work,  recognize  their  talent,  and  highlight 
their  extraordinary  accomplishments  and  discoveries.  To  be  successful  this  journal 
needs  the  entire  community  to  participate  by  submitting  articles,  serving  on  the  editorial 
board,  and  volunteering  as  peer  reviewers.  My  vision  for  the  JDR&E  is  straightforward: 
to  protect,  advocate,  and  serve  as  a  gatekeeper  of  C/CU  generated  by  DoD’s  best  S&T 
investments.  It  will  help  catalyze  new  discoveries  and  world-leading  innovation  to  provide 
our  warfighters  with  the  technologies  that  will  assist  the  Department  in  maintaining  its 
technological  advantage. 


I  would  like  to  take  this  opportunity  to  thank  Ms.  Mary  Miller.  Without  her  vision, 
passion,  and  commitment,  this  journal  would  not  exist.  I  owe  a  tremendous  debt  of 
gratitude  to  her  for  her  guidance  and  support  during  the  publication  of  the  journal’s 
inaugural  issue.  I  am  grateful  for  the  support  of  Mr.  Dale  Ormond  whose  tremendous 
effort  enabled  us  to  connect  with  the  Col  and  DoD  laboratory  community.  I  am  in  debt  to 
Dr.  Gordon  Roesler  from  DARPA  and  the  authors  from  ARDEC  for  graciously  agreeing 
to  contribute  to  the  journal.  I  appreciate  the  assistance  provided  by  my  staff,  Mr.  Roger 
Garay,  Ms.  Michele  Finley,  Mr.  Brent  Ishizaki,  and  Mr.  Jason  Lawrence.  I  also  wish  to 
thank  the  editorial  and  production  staff  for  their  hard  work,  professional  acumen,  and 
devotion  to  the  journal.  Finally,  many  thanks  to  you,  our  readers,  who  join  us  on  this 
journey!  Your  insights  and  feedback  will  help  to  shape  this  journal  into  an  effort  that 
benefits  the  warfighter  -  whom  we  all  support. 
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Reliability  Assessment  of  a  Single-Shot  System  by  Use  of 
Screen  Test  Results 

Abstract:  Field  reliability  prediction  methods  based  upon  early  screening  results 
typically  involve  tracking  a  temporal  metric  such  as  on-time  across  a  constant  stress 
testing  regime  in  order  to  model  wear-out.  These  methods  have  very  limited  applicability 
to  single-shot  systems  because  reliability  is  not  driven  by  wear-out,  and  testing  is  often 
performed  at  varying  stress  levels.  A  new  methodology  is  introduced  to  track  and  project 
the  reliability  for  a  single-shot  system  as  it  goes  through  a  multi-staged  screening  process 
that  produces  no  meaningful  temporal  metric  and  involves  significant  differences  in  test 
strength.  The  approach  described  here  assumes  that  the  defect  density  during  testing 
takes  the  form  of  an  exponential  decay,  although  other  mathematical  functions  can  be 
substituted  for  the  exponential.  In  order  to  apply  the  decay  rate  function  to  a  discrete 
pass/fail  test  scheme,  the  approach  provides  for  normalization  of  the  disparate  tests  to 
constant  stress  by  back-calculating  and  adjusting  for  test  strength  based  upon  previous 
screening  results.  This  approach  is  most  useful  when  reliability  does  not  involve  wear- 
out  of  parts,  which  is  typically  true  of  single-shot  systems.  However,  it  also  potentially 
has  utility  for  all  programs  that  need  to  glean  information  about  early  failures  caused 
by  fabrication  problems,  so  long  as  a  discrete  end  point  for  the  reliability  requirement 
such  as  warranty  termination  has  been  established.  The  equations  provide  a  tool  with 
which  reliability  practitioners  can  estimate  field  reliability  of  a  new  lot  of  single-shot  or 
warrantied  systems  based  upon  early  screen  results,  as  long  as  a  complete  set  of  data  from 
previous  lot  testing  is  available.  Utility  for  reliability  growth  estimation  is  also  described. 
A  numerical  example  is  given  to  demonstrate  application  of  the  model.  This  paper  was 
adapted  from  a  limited-distribution  technical  report  by  the  same  authors  (Coate  and 
Skaggs  2016). 


H.  Coate 
M. Skaggs 
M.  Paulus 
S.  Kamel 

Naval  Undersea  Warfare 
Center  Division 

610  Dowell  Rd,  Keyport,  WA  98345 
mark.paulus@navy.mil 


1.  Introduction 

Modern  single-shot  systems  such  as  guided  munitions 
rely  on  complex  electronic  guidance  and  control  systems. 
Generally,  part  wear-out  can  be  ignored  in  the  service  life 
requirements  of  the  electronics  in  single-shot  systems.  This 
means  that  reliability  is  driven  exclusively  by  what  have 
been  termed  “infant  mortality”  or  “extrinsic”  defects  such 
as  bad  parts  or  poor  workmanship. 

Elimination  of  infant  mortality  defects  in  single-shot 
systems  typically  involves  stress  screening  at  levels  that 
precipitate  failures  without  substantially  reducing  the  life 
of  the  system.  Environmental  Stress  Screening  (ESS)  is 
a  process  or  series  of  processes  in  which  environmental 
stimuli,  such  as  rapid  thermal  cycling  and  random  vibration, 
are  applied  to  electronic  items  in  order  to  precipitate  failure 
of  latent  defects  (Defense  1993).  The  objective  is  to  remove 
as  many  of  the  significant  defects  that  tend  to  dominate  the 
reliability  of  fielded  products  during  early  life.  The  resulting 
product  should  then  more  closely  exhibit  the  inherent  or 
designed-in  reliability  (Bierbaum  2010). 


There  are  quantitative  methods  to  measure  the 
effectiveness  of  an  ESS  program,  such  as  those  identified 
in  (Defense  1993).  These  methods  often  give  the  program  a 
sense  of  the  initial  number  of  patent  and  latent  defects  prior 
to  screening  and  then  the  estimated  latent  defects  that  escape 
to  the  field.  Using  these  results,  the  program  can  get  a  good 
estimate  of  the  field  reliability  of  the  product.  Because  of  the 
physics  of  latent  defect  distribution  in  a  hardware  system, 
the  failures  collected  as  a  function  of  the  amount  of  time 
the  screen  is  applied  resembles  an  exponential  decay  curve 
similar  to  the  front  end  of  the  well-known  “bathtub  curve” 
(Modarres,  Kaminskiy  and  Krivtosv  2010).  After  an  ESS 
program  has  matured,  the  users  will  have  the  parameters  to 
create  the  failure  rate  exponential  decay  curve,  which  allows 
them  to  make  a  reasonable  estimate  of  the  field  reliability 
based  solely  on  the  number  of  patent  defects  precipitated 
during  first  pass  testing  (Defense  1993). 

During  early  tests  of  electronic  systems,  the  circuit 
cards,  interconnects,  and  other  components  that  make 
up  the  subsystem  get  functionally  tested  in  a  binomial 
success/fail  manner.  When  a  failure  occurs  during  one  of 
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these  tests,  fault  isolation  is  performed  and  the  responsible 
component/assembly/CCA  is  fixed  or  replaced  so  that  the 
subsystem  can  move  onto  the  next  test  phase.  For  single¬ 
shot  systems,  the  total  test  duration  bears  no  relationship 
to  actual  usage  of  the  system  and  so  any  specified  temporal 
metric  (on-time/MTBF/etc.)  is  not  recorded  or  tracked. 
Due  to  this,  program-level  decisions  are  usually  based  upon 
the  reliability  metric  of  mission  success  rate  at  the  system 
level  rather  than  a  temporal  metric  such  as  MTBF  at  the 
subsystem  level.  Mission  success  rate  is  easily  defined  and 
measured  during  system  level  testing  (Sherwin  Sep  2009). 
However,  by  the  time  that  those  tests  are  performed  in  any 
significant  quantity,  it  is  generally  too  late  for  corrective 
actions  such  as  design  or  process  changes  to  significantly 
improve  reliability.  This  is  why  there  is  a  clear  need  for  a 
reliability  tool  that  uses  binomial  success  rate  starting  with 
first  pass  test  results  to  provide  pertinent  and  useful  data  to 
assist  the  decision-making  process. 


1 .1  Model  Approach 

A  model  that  tracks  latent  defect  removal  rate  can  show 
where  the  hardware  is  on  the  “bathtub  curve”  (actually 
failure  rate  decay  curve)  when  it  enters  service.  If  the  failure 
rate  has  flattened  out  due  to  removal  of  infant  mortality 
defects,  this  provides  confidence  that  the  field  reliability 
approaches  the  inherent  reliability.  Similar  approaches 
have  been  developed  to  track  the  efficacy  of  reliability 
improvement  efforts.  It  should  be  noted  here  that  defect 
decay  rate,  as  modeled  in  this  paper,  is  conceptually  distinct 
from  reliability  growth  (the  former  develops  a  curve,  the 
latter  looks  to  reduce  the  area  under/over  the  curve). 

A  survey  of  discrete  reliability  growth  models  is 
presented  in  (Fries  and  Sen  1996),  which  consists  of  a 
compilation  of  model  descriptions,  characterizations,  and 
insights.  The  method  of  refining  estimates  of  defect  density 
and  screening  strength  (Procedure  D)  plots  the  failures 
collected  as  a  function  of  the  amount  of  time  the  screen  is 
applied.  The  resultant  plot  is  that  of  an  exponential  decay 
curve.  The  main  difference  between  the  MIL-HDBK-344A 
and  the  discrete  models  is  that  one  is  continuous  and  time 
based,  while  the  other  consists  of  discrete  test  stages.  The 
MIL-HDBK  344A  procedure  D  method  fits  with  the  aim  and 
expectations  of  these  complex  single-shot  test  programs 
(latent  defect  removal  rate  as  a  function  of  test  step). 
However,  there  are  major  differences,  since  procedure  D 
stipulates:  1),  continuous  time  based  versus  discrete  test 
stages,  2),  one  configuration  level,  and  3),  constant  stress. 


The  model  developed  in  this  paper  will  take  discrete 
data  during  sequential  tests  at  different  stresses  and 
configurations  levels  as  an  input  and  normalize  the  data  so 
that  the  methods  similar  to  those  used  in  the  MIL-HDBK- 
344A  can  be  used  to  track  and  project  the  reliability  achieved 
from  screening  of  a  single-shot  system.  The  model  provides 
a  way  to  quantify  both  the  relative  strength  or  effectiveness 
of  each  test  stage,  and  the  effectiveness  of  the  entire  screen. 
The  parameters  of  the  exponential  equation  produced 
by  the  model  can  be  used  to  project  field  reliability  more 
accurately.  The  model  will  also  allow  for  early  test  results 
to  be  used  more  effectively  as  feedback  for  process  or  design 
improvements.  And  lastly,  the  decay  curve  can  be  used  for 
showing  reliability  growth  by  reduction  of  the  area  under 
the  curve. 

The  approach  is  described  in  Section  II,  which  includes: 
1)  methodology;  2)  a  discussion  of  the  data  required;  3)  a  list 
of  model  assumptions;  4)  an  expression  for  determining  test 
equivalency;  5)  an  expression  for  system  failure  counts  at  test 
stage;  6)  a  decay  rate  parameter  estimation  procedure  based 
on  current  lot  data;  and  6)  a  fielded  reliability  projection. 
Section  III  shows  an  example  calculation  for  a  complete 
data  set,  which  includes  an  estimation  of  all  parameters  in 
the  model  and  a  projection  of  the  field  reliability.  Section 
IV  discusses  application  of  the  model.  Concluding  remarks 
and  areas  for  future  work  are  given  in  Section  V. 


2.  Method 

The  method  used  in  this  approach  can  be  described  as 
follows: 

1.  Collect  a  complete  pass/fail  data  set  from  a  prior 
application  of  the  screening  program. 

2.  Apportion  the  discrete  test  steps  into  a  scheme 
that  can  be  normalized  such  that  tests  at  different 
configurations  levels  can  be  used  to  create  a  single 
decay  curve. 

3.  Back-calculate  the  Test  Effectiveness  Factor  (TEF) 
of  each  test  from  the  full  data  set  by  determining 
the  fraction  of  remaining  defects  precipitated  by 
that  test. 

4.  Use  linear  regression  to  estimate  the  key  screen 
parameters  a  and  |3  from  the  graphed  data,  which 
represent  defect  density  of  the  tested  population, 
and  the  overall  strength  of  the  screen,  respectively. 
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5.  Use  the  a  parameter  from  the  next  lot  of  test  data 
to  project  field  reliability. 

6.  With  TEF  and  (3  fixed  by  the  test  program,  measure 
reliability  improvement  from  lot  to  lot  by  tracking 
reductions  in  a. 

7.  Monitor  reliability  growth  by  looking  for 
improvements  in  the  screen  parameters  over 
time  to  detect  improvement  in  the  projected  field 
reliability. 

2.1  Data  Required 

The  projection  model  developed  here  requires  a  specific 
type  of  data.  The  data  that  will  be  normalized  to  system  level 
consists  of  a  complete  set  of  binomial  test  results  at  each  of 
the  independent  test  stages  (ie.  first  past  yield  results  at  the 
card  level  up  to  system  level  field  reliability  and  all  tests  in 
between).  Though  the  individual  test  stages  i=o,i,...,m  can 
be  performed  at  different  configuration  levels  (card  level, 
sub-system,  system)  the  use  of  apportionment  allows  for 
lower  level  tests  to  be  rolled  up  into  system  level  results. 

The  methodology  can  be  applied  to  any  system.  As  an 
example,  suppose  a  system  consists  of  fifteen  cards  divided 
into  three  subsystems  with  five  cards  each  (see  figure  1).  The 
test  program  arbitrarily  declares  fifteen  cards,  one  each  of 
each  type  by  serial  number,  as  being  a  “system  equivalent” 
for  reliability  tracking  purposes.  At  this  point,  there  is  no 
obligation  to  build  the  system  from  those  particular  fifteen 
cards.  The  fifteen  cards  are  tested  at  the  card  level,  and  the 
number  of  functional  failures  is  scored  as  the  initial  data 
point  for  the  reliability  model  (Test  Stage  i=o).  If  three 
functional  failures  occur  on  three  separate  cards  from  the 
population  of  fifteen,  this  is  scored  as  three  system  failures 
for  one  system  level  test.  If  there  were  4  other  sets  of  15  cards 
tested  and  no  other  failures  occurred,  the  total  score  for  that 
test  stage  would  be  3  system  failures  out  of  5  total  system 
level  tests.  The  assumption  made  is  that  the  overall  system 
level  results  at  a  particular  test  stage  will  be  of  binomial 
form.  So  although  there  can  be  multiple  failures  per  system, 
the  overall  distribution  follows  the  binomial  form  where  the 
number  of  failures  in  Ti  trials  is  Ni  -  Binomial  (Ti,  pi)  and 
o<pi. 

After  scoring,  the  three  cards  are  repaired  and  returned 
to  the  population.  At  this  point  the  test  program  may  choose 
to  perform  a  second  card-level  test  such  as  a  HASS  test  (Test 
Stage  i=i).  This  would  be  easily  accommodated  using  the 


same  card-level  equivalency.  Next  the  cards  are  built  into 
subsystems,  although  the  fifteen  cards  may  not  be  used  in  the 
same  subsystem.  The  test  program  now  arbitrarily  declares 
one  each  from  the  three  different  subsystems  as  a  “system 
equivalent”  and  tracking  is  no  longer  at  the  card  level  for 
reliability  purposes  but  rather  at  the  subsystem  level.  The 
subsystems  are  tested  and  then  built  into  actual  systems 
and  equivalency  is  no  longer  necessary.  The  example  within 
this  paper  will  make  this  method  of  apportionment  clear. 

A  system  consists  of  fifteen  cards 

I-  Tests  are  being  r-  _  -  _  _  .  > 


* 

Three  subsystems  with  live  cards  each 


* 


One  system  consisting  of  three  subsystems 


Figure  1  -  Scheme  for  system  equivalency  across  different  testing  levels. 


It  should  be  noted  that  the  apportionment  method 
described  here  necessarily  introduces  some  infidelities.  One 
example  is  that  failures  can  occur  in  a  chassis  that  contains 
interconnects,  but  those  interconnects  are  not  tested  at  the 
card  level  phase.  Another  infidelity  associated  with  testing 
under  environmental  stimulation  (temperature,  vibration, 
etc.)  is  that  some  failures  will  be  precipitated  during  one 
test  phase  but  not  detected  until  a  later  test  phase.  The 
effect  of  these  infidelities  is  largely  mitigated  by  the  use  of 
test  effectiveness  factors  as  described  later  in  this  paper. 

The  main  points  are  that  (1)  any  test  infidelities  are 
of  discrete  magnitude  and  become  less  significant  as  the 
sample  size  grows,  and  (2)  the  infidelities  remain  consistent 
throughout  the  program  and  so  do  not  impact  the  use  of  the 
model  to  show  reliability  improvement  or  deterioration. 
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Through  the  use  of  apportionment,  the  data  will  consist 
of  Ti  and  Ni.  The  numbers  of  trials  Ti,  at  test  stage  i  and 
the  count  data  Ni  for  the  observed  failures  at  that  test  stage 
are  obtained  directly  from  testing.  Once  the  parameters  of 
the  model  have  been  obtained  for  a  test  program,  the  test 
program  must  remain  constant  for  future  projections.  Any 
changes  in  any  of  the  tests  or  the  sequence  of  the  tests  would 
require  a  new  calculation  of  the  model  parameters. 


2.2  Determination  of  Test  Equivalency 


The  proposed  model  allows  a  program  to  take  results 
from  tests  that  occur  at  different  stress  levels  and  different 
system  configuration  levels,  combine  them  and  determine 
an  overall  screen  effectiveness,  and  project  system 
reliability  post  screen.  Since  the  assumed  exponential  form 
results  from  testing  under  constant  stress,  the  individual 
test  stages  that  make  up  the  screen  must  be  normalized 
to  constant  stress.  This  is  accomplished  by  calculating 
the  relative  effectiveness  of  each  test  against  the  entire 
screen.  This  relative  effectiveness  will  be  referred  to  as  Test 
Effectiveness  Factor  (TEF).  The  TEF  of  an  individual  test 
stage  is  given  by 


where, 


(1) 


Nt 

hi  = 
a  Ti 


\,2 


m+1M 

-Ti 


l=t 


(2) 


(3) 


A_(i,i)  is  the  total  number  of  failures  during  the  test 
stage,  and  X_(i,2)  is  the  total  number  of  failures  that  will 
occur  through  the  entire  test  program  beginning  with  the 
current  test  stage. 


The  expression  for  TEF  found  in  Eq.i  is  similar  to 
an  expression  developed  by  the  Army  Materiel  Systems 
Analysis  Activity  (AMSAA)  for  Fix  Effectiveness  Factor 
(FEF)  during  the  Test-Analyze-and-Fix  (TAAF)  process 
(PM,  WJ  and  WJ  2000).  The  FEF  expression  is  used  to 
calculate  the  effectiveness  of  an  individual  fix  as  a  remedy 
to  a  particular  failure  mode.  In  the  model  described  in  this 


paper,  the  TEF  expression  found  in  Eq.i  is  used  to  assess 
the  effectiveness  of  a  test  phase  at  reducing  the  latent  defect 
density  of  a  population.  Each  test  phase  within  the  screen 
reduces  the  fraction  of  latent  defects  within  the  population 
by  a  different  amount  and  the  magnitude  of  the  fraction 
reduction  is  directly  related  to  the  strength  of  the  individual 
test.  The  strength  of  a  test  phase  is  obtained  from  the  ability 
to  both  precipitate  failures  and  then  detect  those  failures. 
This  typically  results  in  a  HASS  test  having  a  higher  TEF 
than  that  of  a  bench  level  functional  test.  For  additional 
information  on  the  application  of  the  TEF  see  the  example 
calculation  in  section  III. 

2.3  Model  Assumptions 

1.  Test  trials  at  the  card  and  subsystem  level  have 
the  potential  for  one  or  more  failures  at  the  system 
level  such  that  total  failures  per  system  test  can  be 
larger  than  1. 

2.  Each  test  trial  at  the  system  level  results  in  a 
dichotomous  success/failure  outcome  such  that 
Ni,j  -  Bernoulli  (pi)  for  each  i=o,i,...,m  and 
j=i,-,T. 

3.  The  distribution  for  the  number  of  failures  in  T 
trials  at  system  level  test  stages  is  binomial  such 
that  Ni  ~  Binomial  (T,  pi)  for  each  i=o,i,...,m. 

4.  The  field  reliability  is  binomially  distributed  with 
probability  p  of  success. 

5.  Since  the  model  does  not  use  any  temporal  metrics, 
very  weak  tests  can  be  combined  with  either  the 
prior  or  the  subsequent  test  and  considered  to 
be  a  single  test.  An  example  where  this  would 
be  of  value  is  if  HASS  vibration  results  in  a  TEF 
lower  than  0.05,  but  it  is  determined  that  the 
HASS  temperature  test  immediately  following  has 
failure  events  that  might  have  been  precipitated 
because  of  the  combination  of  the  sequence  of  the 
vibration  and  temperature  environments.  The 
new  test  stage  would  consist  of  both  tests  with  N 
being  the  addition  of  the  failure  counts  from  each. 

6.  The  TEF  for  the  field  environment  is  assumed  to 
be  equal  to  1.  This  is  in  keeping  with  the  intended 
use  of  a  single-shot  system  in  which  the  first  field 
test  is  the  final  test  of  interest.  Alternatively  for 
continuous  use  systems,  the  TEF  can  be  set  at  1  at 
the  termination  of  a  warranty  as  long  as  no  wear- 
out  failures  are  anticipated. 
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2.4  Estimation  Of  Failure  Counts  At 
Test  Stage 

As  described  earlier,  TEF  adjustments  can  be  used 
to  normalize  disparate  tests  to  approximate  constant 
stress.  Under  constant  stress,  the  number  of  defects  in  a 
population  during  early  screening  has  been  shown  to  follow 
the  basic  exponential  decay  expression  y=ae_(3t  by  both 
theoretical  and  empirical  methods  (Defense  1993)  (Fries 
and  Sen  1996)  (H.H.  and  C.P.  1992).  At  the  conceptual 
level,  this  can  be  thought  of  as  a  rapid  narrowing  of  the 
window  of  opportunity  for  a  defect  to  survive  one  test  but 
accumulate  enough  damage  to  fail  the  next  test  at  the  same 
stress  level.  This  means  that  when  the  TEF  adjustment  is 
applied  to  failure  count  data  from  individual  test  phases 
of  a  screening  program,  the  shape  of  the  resultant  defect 
density  plot  should  approximate  an  exponential  decay 
(figure  1).  This  is  consistent  with  the  models  developed  and 
employed  by  (Defense  1993)  and  (Fries  and  Sen  1996).  The 
exponential  decay  of  defects  within  a  population  and  overall 
hazard  rate  during  exposure  to  a  constant  stress  in  early  life 
is  similar  to  the  infant  mortality  region  of  the  bathtub  curve 
(Modarres,  Kaminskiy  and  Krivtosv  2010)  and  is  prevalent 
throughout  reliability  literature.  A  full  justification  for  the 
use  of  an  exponential  decay  shape  is  beyond  the  scope  of 
this  paper,  but  a  thorough  treatment  can  be  found  in  (H.H. 
and  C.P.  1992).  While  the  cited  literature  is  followed  in 
this  paper  with  the  assumption  of  an  exponential  shape, 
circumstances  may  dictate  that  a  different  function  be  used 
with  the  described  approach.  In  those  cases,  the  discussion 
found  later  in  this  paper  of  the  exponential  parameters  a 
and  (3  would  not  apply. 


Test  Stage 


Figure  2  -  Conceptual  model  of  relationship  between  defects  removed 
and  defects  remaining  showing  the  exponential  decay  shape. 


The  equations  found  in  (4)  and  (5)  are  nearly  identical 
to  the  discrete  exponential-growth  model  detailed  in  (Fries 
and  Sen  1996).  The  only  difference  is  the  addition  of  the 
TEF  adjustment  term  di  into  the  equation  to  normalize  the 
stress  level.  This  constant-stress  test  performed  multiple 
times  is  what  Lloyd  &  Lipow  were  modeling  in  (Fries  and 
Sen  1996)  and  is  also  the  basis  for  most  Discrete  Reliability 
Growth  Models  (DRGM)  (PM,  WJ  and  WJ  2000). 

The  parameters  found  in  (5)  directly  correlate  to  specific 
aspects  of  a  testing  effort.  The  di  term  represents  the  test 
effectiveness  of  the  individual  test  stage  i.  The  calculations 
of  di  for  each  test  are  done  after  a  complete  data  set  has 
been  obtained.  With  the  TEF  adjustments  applied,  the 
other  parameters  within  the  model  can  be  estimated.  The 
a  parameter  is  related  to  the  defect  density  of  the  particular 
lot  of  items  being  tested.  The  (3  parameter  correlates  to  the 
overall  effectiveness  of  the  series  of  tests  at  screening  out 
the  defects  and  will  be  referred  to  as  the  stress  constant  of 
the  screen. 


Ultimately  the  goal  of  any  screening  program  is  to  ensure 
the  items  exiting  the  program  are  past  the  exponential  decay 
portion  of  the  “bathtub”  curve  and  into  the  random  failure 
portion.  Using  the  TEF  adjustment  from  equation  (1),  the 
failure  count  expression  obtained  is  given  by 

Nj  =  T;  ■  dj  •  a  •  e~p‘’  (4) 

From  the  expression  found  in  Eq.4  one  can  also  obtain 
the  expression  for  the  failure  probability  for  an  individual 
trial: 


Pi  =  ^  ■  a  ■  e 


-P-1 


(5) 


2.5  Parameter  Estimation  And  Field 
Reliability 

As  previously  mentioned,  empirical  determination 
of  the  parameters  requires  a  complete  data  set  from  the 
application  of  the  screen  and  at  least  some  results  from  field 
testing  (m+i  test  stage).  Once  the  TEF  (d.)  for  each  test 
stage  is  found,  the  a  and  (3  can  be  estimated.  The  estimation 
procedure  for  a  and  (3  after  data  collection  is  done  by 
rearranging  equation  (4)  to  the  following  expression.  After 
taking  the  natural  log  of  each  side,  the  expression  takes  on 


Nt  1 

—  ■~r  =  a 


>~P'i 


(6) 


the  form  of  the  familiar  linear  equation  of  y  =  mx  +  b,  where 
the  left  side  of  equation  (7)  is  y;  In  a  =  b;  -(3  =  m;  and  i  =  x. 
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WV:  1\ 

(nVT  '(f)  =  lna~  P  '*  (7) 

With  equation  (4)  in  the  form  of  (7),  the  failure  count 
data  of  Ni  and  Ti  at  the  individual  stages  of  i=o,  1,...,  m+i 
can  be  plotted.  Then  by  applying  a  least-squares  regression 
analysis,  a  and  (3  can  be  found.  The  resultant  calculations 
for  di,  a,  and  (3  are  useful  on  the  lot  that  was  just  tested 
to  determine  the  effectiveness  of  the  entire  screen,  get 
an  estimate  on  the  initial  defect  density,  and  also  make 
additional  estimations  on  the  fielded  reliability  as  well  as 
remaining  defect  density.  Immediately  following  the  screen, 
estimating  the  probability  of  success  in  the  field  from  the 
screen  data  would  be  done  by  extending  the  expression  in 
(5)  to  the  i=m+i  case  and  subtracting  the  result  from  1.  This 
expression  can  be  found  below. 

Rm+ 1  =  1  —  a  •  (8) 

Once  calculations  have  been  made  for  di  and  (3,  and 
since  they  are  only  dependent  on  the  screen  being  applied, 
they  can  be  held  constant  for  the  next  lot  of  test  items  to  go 
through  the  screen.  This  means  that  the  only  parameter  left 
to  be  estimated  is  a  which  is  related  to  the  defect  density 
of  the  current  lot.  As  a  program  matures,  the  reliability 
growth  of  that  program  will  be  reflected  by  a  decreasing  a 
from  lot  to  lot. 

2.6  Model  Applied  to  Synthetic 
Test  Data 

The  data  set  in  the  following  example  is  representative 
of  data  that  comes  from  a  screen  applied  to  an  electronic 
sub-assembly  that  is  responsible  for  the  control  of  a  single¬ 
shot  guided  munition.  The  sub-assembly  will  be  discussed 
in  general  terms  and  the  failure  data  provided  will  be 
simulated  but  representative  of  data  gathered.  The  example 
assumes  that  the  second  lot  is  entering  the  manufacturing 
phase  before  testing  has  been  completed  on  the  first  lot. 
On  a  lot  of  175  systems,  all  have  been  through  the  system 
functional  test,  but  only  70  have  had  a  field  test,  and  only  10 
have  been  used  in  the  field. 

The  screening  program  that  the  control  system  goes 
through  before  being  released  for  use  is  as  follows;  Stage 
o)  Full  functional  bench  level  testing  at  the  card  level,  Stage 

1)  Full  functional  testing  at  the  Sub-Assembly  Level,  Stage 

2)  HASS  Vibration  testing  at  the  Sub-Assembly  level,  Stage 


3)  HASS  Temperature  testing  at  the  Sub-Assembly  level, 
Stage  4)  Full  Functional  testing  at  the  System  Level,  Stage 
5)  Field  environment  full  functional  system  level  test.  The 
representative  test  data  from  this  sequence  of  tests  is  seen 
in  Table  1  and  in  graphical  form  as  failures  per  test  in  Figure 
3.  The  data  shown  in  Table  1  and  Figure  3  is  reported  at 
the  Sub-Assembly  level  following  the  application  of  the 
apportionment  method  described  in  the  “Data  Required” 
section  of  this  paper. 


Table  1  -  Screening  Program  Results 


Test 

Stage  (i) 

Test 

Description 

Configuration 

Failures 

(N.) 

Total 

Tested  (T.) 

0 

1st  Functional 

Card  Level 

71 

175 

1 

F  unctional 

Sub- Assembly 

3 

175 

2 

HASS  Vibe 

Sub- Assembly 

12 

175 

3 

HASS  Temp 

Sub- Assembly 

65 

175 

4 

Functional 

System 

18 

175 

5 

Field  Test 

System 

16 

70 

6 

Field 

System 

1 

10 

Figure  3  -  Unadjusted  failures  per  test  as  a  function  of  the  test  stage 


Note  that  there  is  not  a  clear  trend  of  decreasing  failure 
rate  in  figure  3.  This  is  due  to  the  varying  levels  of  stress 
applied.  Because  the  approach  described  in  this  paper  is 
not  temporally  based,  there  is  flexibility  in  deciding  the 
beginning  or  end  of  an  individual  test.  In  the  data  shown 
above,  the  first  card  level  test  is  obviously  important  in 
precipitating  infant  mortality  defects.  The  second  test,  first 
functional  at  the  subassembly  level,  looks  relatively  weak. 
The  second  test  is  basically  a  repeat  of  multiple  card  level 
tests,  but  on  a  full  subassembly  and  including  verification 
of  interconnect  integrity  between  boards  and  chassis  that 
could  not  be  accomplished  at  the  card  level.  The  second 
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test  can  be  combined  with  the  first  test  to  form  a  single  test 
for  scoring  purposes.  While  this  is  not  necessary  for  the 
model,  it  avoids  an  undesirable  situation.  With  only  three 
failures  during  test  stage  l,  the  TEF  will  be  low  (0.019  in  the 
example),  meaning  that  the  adjustment  will  be  large.  This 
makes  this  ineffective  test  very  sensitive  to  every  additional 
failure,  such  that  a  single  failure  will  increase  the  failure  rate 
33%.  The  result  will  be  such  that  even  after  the  exponential 
decay  has  stabilized  for  the  test  program,  this  number  will 
still  be  jumping  around.  This  is  obviously  undesirable  for 
tracking  improvements. 

A  similar  situation  arises  with  the  third  and  fourth  test. 
The  HASS  vibration  and  HASS  temperature  tests  are  shown 
separately,  with  a  low  failure  rate  on  the  vibration  test. 
Work  has  shown  that  failures  precipitated  by  vibration  will 
often  not  be  detected  until  the  assembly  experiences  rapid 
temperature  swings,  primarily  because  continuity  may 
still  exist  across  a  broken  solder  joint  if  the  two  sides  are 
in  casual  contact.  This  means  that  the  failure  rates  of  the 
two  tests  are  interdependent  and  so  can  be  combined  into  a 
single  test  for  scoring  purposes.  The  revised  test  results  can 
be  seen  in  Table  2. 


Figure  4  -  The  revised  plot  of  failure  rate  as  a  function  of  the  test 
stage  after  combining  ineffective  tests  with  more  effective  tests  that  is 
sequentially  adjacent. 


Tl(2 


-Zb 
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'  ?! 
77 


77 


d,  = 


^t,2 


■  =  —  =  0.440 

(9) 
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18  16  1 

4  4  Wn  4  7T  —  0.871 

175  70  10 

(10) 

0.440 

— — —  =  0.505 

0.871 

(11) 

Table  2  -  Screening  Program  Revised  Results 


Test 

Description 

Failures 

(Nj) 

Total 

Tested  (T.) 

O 

1st  Functional 

74 

175 

1 

HASS 

77 

175 

2 

Functional 

18 

175 

3 

Field  Test 

16 

70 

041 

Field 

1 

10 

Note  that  there  is  still  no  clear  trend  of  decreasing 
failure  rate  in  figure  4. 

2.7  Calculation  And  Application  Of 
TEF  Adjustment 

In  order  to  calculate  the  TEF  adjustment  of  each  test 
stage  it  is  necessary  to  have  a  complete  data  set  like  that 
shown  in  Table  2.  Then  using  expressions  (i)-(3),  the  TEF 
at  each  stage  can  be  calculated  independently.  An  example 
calculation  for  test  stage  1  from  Table  2  can  be  seen  below. 


The  TEF  adjustments  for  stages  i=o,  2,  and  3  are  easily 
calculated  in  the  same  manner  as  above.  With  the  TEF 
adjustments  calculated  for  each  test  stage  one  can  begin  to 
calculate  the  values  on  the  left  side  of  equation  (6).  These 
values  represent  the  normalized  failure  rate  of  each  test. 
These  values  can  also  be  thought  of  as  the  defect  density  of 
each  item  prior  to  that  test  stage.  In  other  words,  if  the  test 
at  stage  i  was  100%  effective  it  would  remove  on  average 
this  amount  of  defects  from  each  assembly.  These  values 
can  be  seen  in  Table  3. 

The  graph  in  Figure  5  shows  a  plot  of  the  defect  density 
versus  test  stage  after  the  TEF  adjustment  is  applied. 
The  figure  illustrates  the  exponential  decay  of  the  overall 
defect  density  as  a  function  of  test  stage.  While  the  defects 
removed  (failures/test  as  in  Figure  4)  at  each  test  does  not 
have  a  shape  consistent  with  a  parametric  model,  when 
one  applies  the  TEF  adjustments  and  plots  the  total  defect 
density,  a  shape  consistent  with  that  of  an  exponential  decay 
becomes  apparent.  The  right  side  of  the  expression  in  (6) 
and  the  parameters  a  and  (3  that  make  up  that  expression 
are  used  to  model  this  exponential  decay 

N-  1 

Dj=  -=r'-r,  left  side  of  expression  (6) 

di 
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Note  that  after  the  normalization  based  upon  test 
strength,  the  defect  density  decay  rate  follows  an  exponential 
decay. 


With  the  a  and  (3  calculated  one  could  then  make  the 
point  estimate  for  the  probability  of  failure  of  an  individual 
field  event  using  expression  5. 


Table  3  -  Application  of  the  TEFs  result  in  estimation  of  defect  density. 


Test 
Stage  (i) 

TEF  (d.) 

Configuration 

Defect 

Density  (Di) 

0 

0.327 

Card  Level 

1.294 

1 

0.505 

Sub-Assembly 

0.871 

2 

0.238 

Sub-Assembly 

0.431 

3 

0.696 

Sub-Assembly 

0.329 

4 

1 

System 

0.100 

Ln(DD)  Linear  Regression 


Figure  6  -  Linear  Regression  of  the  Ln(Defect  Density)  versus  test  stage. 


Pm+1  =  a  ■  e"^m+1)  =  1.48e“6096'4  =  0.1292  (15) 

Note  that  while  the  field  data  showed  90%  reliability 
in  a  sample  of  ten,  the  model  shows  that  the  actual  field 
reliability  will  be:  1-0.1292=0.871  or  87.1%  reliability. 


Figure  5  -  Plot  of  the  defect  density  versus  test  stage  after  normalizing 
for  test  effectiveness. 


2.8  Parameter  Estimation  Thru  Least 
Squares  Linear  Regression 

With  the  TEFs  calculated  and  the  resultant  plot  of  defect 
density  complete,  one  can  now  estimate  the  remaining 
parameters  of  a  and  (3  with  expressions  (4)-(8).  The  simplest 
way  to  do  this  is  by  plotting  the  natural  log  of  the  defect 
density  (left  side  of  equation  7)  against  the  test  stage  i.  A 
least  squares  linear  regression  curve  fit  can  then  take  place 
on  the  plot  (Figure  6).  Using  the  resultant  linear  equation 
one  can  solve  for  a  and  (3,  where  the  relationship  of  a  and  P 
to  the  straight  line  equation  is  described  in  expression  (7). 
The  calculations  of  a  and  P  can  be  seen  below: 

y  =  —0.6096%  +  0,3921 
p  =  0.6096 

In  a  =  0.3921  ->  a  =  e 0  3921  =  1.4 


(12) 

(13) 

(14) 


3.  Results  &  Discussion 

With  the  model  parameters  that  describe  the  test 
program  estimated,  when  the  next  lot  of  items  come  into  the 
testing  program,  one  can  start  to  analyze  the  results  using 
the  model  to  determine  the  likelihood  that  the  lot  will  meet 
the  field  reliability  requirement.  The  program  can  also  set 
requirements  at  the  early  test  stages  that  the  program  needs 
to  achieve  in  order  to  realize  the  field  reliability  requirement. 
The  first  step  in  application  of  the  model  is  to  solve  for  the 
desired  maximum  failure  rate  at  each  test  stage.  These  can 
be  solved  by  rearranging  expression  5  in  order  to  solve  for 
amax  using  the  field  requirement  for  pi,  di=i,  P,  and  i=m+i. 
Included  below  is  the  example  calculation  for  the  data  set 
of  this  paper  with  the  field  requirement  set  to  0.05  failure 
probability. 


_  Pm+1 
amax  ~  e-/j.(m+i) 

0.05 

amax  -  - -0.6096  (4)  —  0-5727 


(16) 

(17) 
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Now  that  amax  is  solved  for,  the  maximum  failure 
probabilities  at  each  test  stage  can  be  calculated  by  using 
expression  5  and  the  TEFs.  These  values  for  the  synthetic 
test  data  set  are  found  in  Table  4. 


Table  4  -  Maximum  allowable  Failure  Rates  at  each  Test  Stage 


Test  Stage 
0) 

Models  Max  allowable  Pr(Failure) 

0 

0.1871 

1 

0.1572 

2 

0.0403 

3 

0.0639 

When  compared  to  the  actual  test  results  that  this  data 
was  generated  from  these  values  are  all  lower  than  what  was 
realized.  This  makes  sense  due  to  the  fact  that  the  model’s 
actual  projection  (from  the  test  data)  for  field  probability  of 
failure  was  0.1292.  So  in  order  to  get  that  projection  down 
to  0.05  the  requirement  at  each  test  stage  is  stiffer  than  what 
was  experienced.  With  the  limits  set,  when  test  results  start 
to  come  in  they  can  be  used  to  calculate  the  probability  that 
the  requirement  at  that  stage  is  going  to  be  met  when  all 
of  the  items  have  been  received.  The  details  of  this  type  of 
calculation  are  not  covered  in  this  paper,  but  are  described 
in  (Modarres,  Kaminskiy  and  Krivtosv  2010)  and  (O’Connor 
2011).  Also  when  test  data  comes  in,  calculations  can  be 
made  towards  solving  for  the  a  of  the  current  lot.  Once  an 
estimate  for  the  a  of  the  current  lot  is  made,  a  projection 
for  field  reliability  can  also  be  realized.  This  projection  can 
provide  a  program  with  a  useful  tool  early  in  a  production 
lot  to  assess  reliability. 

4.  Conclusion 


hardware.  Similarly,  when  the  manufacturing  process  and 
the  hardware  are  stable,  the  approach  can  gauge  the  effect 
of  changes  in  the  test  protocol. 

While  the  emphasis  here  has  been  on  single-shot 
systems,  the  use  of  a  latent  defect  decay  rate  curve  and  test 
effectiveness  factors  has  wider  applicability  to  reusable 
systems  that  have  a  defined  end-of-life  requirement 
such  as  reaching  the  end  of  a  warranty  period  with  a 
desired  reliability.  The  approach  can  be  applied  in  any 
circumstances  where  the  final  testing  phase  can  be  assigned 
aTEF  of“i.” 

5.  Future  Work 

The  primary  limitation  of  the  described  model  is  the 
need  for  a  full  data  set  to  determine  the  exponential  decay 
parameters.  This  limitation  can  be  mitigated  somewhat  by 
the  use  of  Bayesian  predicting  based  upon  other  systems 
that  were  tested  in  a  similar  manner.  A  Bayesian  approach 
may  be  particularly  well  suited  to  situations  where  the  test 
sequence  changes  during  tracking.  Also,  estimation  of  TEF 
could  be  accomplished  with  a  Monte  Carlo  approach. 
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